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Since its inception, the 21st Century Community Learn¬ 
ing Centers (CCLC) program has been one of the 
fastest-growing federal programs. In 2008, its budget of 
over $1 billion funded 9,930 centers (U.S. Department 
of Education, 2008). The program provides grants to 
schools and community organizations to expand educa¬ 
tion services beyond the regular school hours. The tuto¬ 
rial services and academic enrichment activities of 21st 
CCLC programs are thus often designed to help youth 
meet local and state academic standards in subjects 
such as reading and math. 

Because there are so many 21st CCLC programs, 
accountability systems and impact studies have become 
an important focus. However, research findings have 
been mixed. While some studies find a positive associa¬ 
tion between afterschool participation and the develop¬ 


ment of academic and social skills (Cosden, Morrison, 
Albanese, & Macias, 2001; Huang, Gribbons, Kim, Lee, 
& Baker, 2000; Klien & Bolus, 2002; Mahoney, Lord, & 
Carryl, 2005; Posner & Vandell, 1994; Welsh et al., 
2002), other research has found no effects or has found 
negative associations between achievement and after¬ 
school participation (Bissell, Dugan, Ford-Johnson, & 


JENELL HOLSTEAD received her doctoral degree in school psychol¬ 
ogy from Indiana University in April 2010. She has worked at the 
Center for Evaluation and Education Policy since 2007 and has re¬ 
cently accepted a faculty research associate position there. She has 
served as a local evaluator for a number of 21st Century Commu¬ 
nity Learning Centers programs in Indiana. She is an integral team 
member of the statewide evaluations for the 21st CCLC programs in 
Indiana and Kentucky. 

MINDY HIGHTOWER KING holds a faculty research appointment 
at the Center for Evaluation and Education Policy, where she designs 
and manages evaluation projects for education and human services 
organizations. Her work focuses on evaluating and enhancing the 
quality of 21st CCLC afterschool programs and on building the 
capacity of charter schools and state departments of education to 
enhance accountability by implementing performance measurement 
systems. She currently serves as the project director for the statewide 
evaluations of the 21st CCLC initiatives in Indiana and Kentucky. 











Jones, 2002; James-Burdumy, Dynarski, & Deke, 2007; 
National Institute of Child Health and Human 
Development Early Child Care Research Network, 2004; 
Pettit, Laird, Bates, & Dodge, 1997; U.S. Department of 
Education, 2003). The discrepancies maybe due to sam¬ 
ple characteristics, most notably socioeconomic status 
(Marshall et al., 1997; Posner & Vandell, 1994), and to 
lack of methodologic rigor in many evaluations of after¬ 
school programs (Scott-Little, Hamann, & Jurs, 2002). 

Recently the focus of research on afterschool pro¬ 
grams has shifted to better understanding which programs 
are successful in effecting academic gains in students and 
why (Beckett et al., 2009). This research has suggested a 
number of program practices that help students make aca¬ 
demic gains. Assessment procedures can help afterschool 
programs increase the effective implementation of these 
practices and can inform ongoing improvement efforts. 

External assessments, often conducted by third-party 
organizations specifically trained in evaluation, can pro¬ 
vide program staff with an outside perspective on program 
quality. These assessments generally require significant fi¬ 
nancial resources and tend to focus on outcomes such as 
grades, test scores, and survey data. They often place less 
emphasis on the practices that influence the afterschool 
environment and the program activities youth experi¬ 
ence—the “point of service” aspects of afterschool quality 

Self-assessment is an often-overlooked alternative to 
external assessment. Program staff can use self-assessment 
processes to systematically review the quality of their after¬ 
school programming and to facilitate discussions on ways 
to enhance it. Self-assessment of point-of-service activities, 
which can provide a wealth of valuable information regard¬ 
ing program quality (Akiva & Smith, 2007), should be 
used regularly to enable ongoing program improvement. 

Despite the fact that researchers have reached some 
consensus regarding point-of-service elements that sup¬ 
port student academic gains, most self-assessment tools 
for afterschool programs still do not consistently include 
these elements. Because a major goal of 21st CCLC pro¬ 
grams is to improve participants’ academic performance, 
this gap in self-assessment tools can leave programs with¬ 
out essential information on program quality, even when 
they use such tools frequently. This article reviews pro¬ 
gram elements shown to effect academic growth and ex¬ 
amines the extent to which available self-assessment tools 
measure these practices. None of the available self-assess¬ 
ment tools measures the extent to which programs include 
all of the point-of-service elements that support academic 
enrichment. Therefore, standardized self-assessment tools 
for afterschool programs should be enhanced or devel¬ 
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oped to include evidence-based practices known to be ef¬ 
fective in improving academic achievement. 

Features of High-Quality Programs That 
Contribute to Academic Outcomes 

Self-assessment instruments to be used by programs 
seeking to improve academic achievement should mea¬ 
sure the program practices and characteristics that have 
been linked to students’ academic success. In the past 10 
years, a number of studies have attempted to discover 
what these practices are (Farber, 2007; Fashola, 2005, 
Hartry, Fitzgerald, & Porter, 2008; Lauer et al., 2006; 
Miller & Hall, 2007; Roth, Brooks-Gunn, Murray, & 
Foster, 1998; Shumow, 2001). The studies have identi¬ 
fied a number of structural components and capacity ele¬ 
ments that are associated with increased academic 
achievement, including supportive and educated staff, 
environments in which children can learn new skills and 
exercise choice, adequate resources, and good relation¬ 
ships with school personnel. Point-of-service practices 
observed in programs that were successful in helping 
students make academic gains included offering home¬ 
work help, providing one-on-one tutoring, and linking 
afterschool activities to the school day. 

More recently, the U.S. Department of Education’s 
Institute for Education Sciences (IES) developed a prac¬ 
tice guide, Structuring Out-of-School Time to Promote 
Academic Achievement (Beckett et al., 2009), which in¬ 
cludes a set of recommendations for afterschool pro¬ 
grams to help students benefit academically. The guide, 
developed by a panel of experts in out-of-school time 
programs who examined high-quality experimental and 
quasi-experimental studies to identify program practices 
associated with positive academic outcomes, includes 
only those practices that were supported by adequate 
levels of empirical evidence to warrant broad-based rec¬ 
ommendations. The guide’s five recommendations in¬ 
form our examination of self-assessment tools. They are: 

• Aligning the out-of-school time program academically 
with the school day 

• Maximizing student participation and attendance 

• Adapting instruction to individual and small-group 
needs 

• Providing engaging learning experiences 

• Assessing program performance and using the results 
to improve program quality (Beckett et al., 2009) 

Of the practice guide’s recommendations, we exam¬ 
ine only elements that occur at the point of service, when 
youth are participating in activities at the program site 
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(Granger, Durlak, Yohalem, & Reisner, 2007), as these are 
the elements that can be evaluated through observation 
and self-assessment. In addition, point-of-service prac¬ 
tices are particularly important because they directly af¬ 
fect youths’ decisions to attend the program and the ben¬ 
efits participants gain. The last recommendation of the 
practice guide, regarding using data to improve the pro¬ 
gram, is not an observable point-of-service element, but it 
could be addressed by the self-assessment process itself 

Align the Out-of-School Time Program 
Academically with the School Day 

Typically, effective afterschool programs directly and pur¬ 
posefully connect academic program components to the 
school day (Policy Studies Associates, 1995). In fact, the 
IES practice guide suggests that aligning the afterschool 
program with the school day is a necessary component of 
academic improvement (Beckett et al., 2009). Academic 
alignment can often be directly observed at the point of 
service delivery. 

Coordinating curriculum is one example of this align¬ 
ment. To achieve coordination, afterschool staff may do 
one or more of the following: use school curricula directly, 
provide homework assistance and activities that promote 
basic skills, or develop activities consistent with district 
and state learning standards. Afterschool programs can re¬ 
inforce critical skills and knowledge by offering activities 
that complement, but are different from, school activities. 

Another way to achieve alignment is frequent and 
ongoing communication between school and afterschool 
staff. Teachers’ information about school day instruction 
or individual student needs can help afterschool staff 
plan programming (Beckett et al., 2009). The communi¬ 
cation can occur both informally, such as in hallway con¬ 
versations, and formally in regularly scheduled meetings 
and ongoing updates on students’ progress. 

Maximize Student Participation and Attendance 

Many studies have demonstrated that students who par¬ 
ticipate in afterschool programs frequently and for longer 
periods of time are more likely to demonstrate social and 
academic benefits than those who do not (Department of 
Education, University of California at Irvine, 2001; 
Eluang et al., 2000; Johnson & Jenkins, 2000; Welsh et 
al., 2002). For example, the evaluation of LA’s BEST pro¬ 
gram, conducted over a ten-year period, found that regu¬ 
lar attendance of at least 150 days per year for more than 
one year was necessary for positive impact on academic 
performance. The highest gains in standardized math, 
reading, and language arts scores were found in students 


with four years of regular participation (Department of 
Education, University of California at Irvine, 2001). The 
evaluation also found that regular attendance over mul¬ 
tiple years was related to better school attendance, in¬ 
creased engagement in school, and higher aspirations to 
finish school and go to college (Huang et al., 2000). 

To maximize student participation and attendance, 
Beckett and colleagues (2009) suggest examining impor¬ 
tant factors including location, program offerings, trans¬ 
portation, timing, length, and frequency of services. 
Programs should be easily accessible and convenient for 
youth; program offerings should be engaging and interest¬ 
ing. In addition, addressing the changing developmental 
needs of older youth helps to keep them engaged in after¬ 
school programming (Deschenes, Little, Grossman, & 
Arbreton, 2010). Program staff should assess the extent to 
which students attend the program and the level of en¬ 
gagement students exhibit at the point of service delivery. 

Adapt instruction to Individual and 
Small-Group Needs 

Individualized instruction is an important feature in im¬ 
proving student performance (Lauer et al., 2004). Because 
afterschool program time is significantly shorter than the 
school day, instruction must be focused and targeted 
(Beckett et al, 2009). Afterschool instruction and activi¬ 
ties must therefore be adapted to meet the needs of indi¬ 
vidual participants. Instruction that is aligned and paced 
to individual student needs results in improved academic 
performance (Slavin, 2006). To provide individualized in¬ 
struction, program staff must be aware of each child’s 
strengths and weaknesses (Beckett et al., 2009). An activ¬ 
ity that is not cognitively stimulating for some students 
could be too difficult for others. Program staff should 
therefore use formal and informal assessment data to learn 
what adaptations are necessary (Beckett et al., 2009). 

A number of studies have reported that after¬ 
school programs that affect academic performance 
provide opportunities for students to interact in small 
groups (Rosenthal & Vandell, 1996) or offer one-on- 
one tutoring support (Baker, Gersten, & Keating, 
2000; Beckett et al., 2009). Lauer and colleagues 
(2004) reported that one-on-one tutoring improved 
the reading levels of at-risk students. In addition, 
Zuman and Miller’s (2005) evaluation of afterschool 
programs in Massachusetts found that program qual¬ 
ity was closely linked with small group sizes and low 
student-to-staff ratios. Taken together, these studies 
suggest that afterschool programs should use one-on- 
one or small-group tutoring to provide targeted as- 
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sistance to students who need help beyond what they 
receive during the school day (Beckett et al., 2009). 

Provide Engaging Learning Experiences 

Studies indicate that programs that improve student per¬ 
formance provide highly engaging activities that incor¬ 
porate academic content (August, Realmuto, Hektner, & 
Bloomquist, 2001; Borman, Goetz, & Dowling, 2008; 
Karcher, Davis, & Powell, 2002). Afterschool programs 
typcially offer a diverse blend of academic pursuits, fine 
arts and crafts, and physical or recreational activities. 
Although offering such variety is considered to be best 
practice (Fashola, 1998; Rosenthal & Vandell, 1996; 
Vandell et ah, 2004) and appears to 
be important in capturing youth 
interest and maintaining involve¬ 
ment, research demonstrates that 
games, recreation, and field trips 
are ineffective in improving aca¬ 
demic performance when they are 
independent ol the academic com¬ 
ponent of the program (U.S. 

Department of Education, 2008). 

In order to ensure gains in student 
achievement, the engaging activi¬ 
ties that interest and motivate students must be explicitly 
connected to academic learning activities (Beckett et ah, 
2009). 

Another essential ingredient of improved academic 
outcomes is active learning (Fredricks, Blumenfeld, & 
Paris, 2004). Specifically, programs must actively engage 
children by generating opportunities to practice new 
skills through hands-on experiences, practical examples, 
cooperative learning experiences, and real-world activi¬ 
ties; instruction must be connected to student interests 
(Beckett et al., 2009). In fact, Noam (2003) concludes 
that in order to avoid “having the children and staff expe¬ 
rience [afterschool] projects just as more school,” after¬ 
school programs should strive to serve “as a creative ex¬ 
tension of learning that is more hands on, more 
participatory, and more community-focused” (p. 136). 
Afterschool program activities should involve fewer 
large-group lecture activities and more opportunities for 
youth to engage in hands-on learning activities. 

Review of Standardized Self-Assessment Tools 

The many available standardized self-assessment tools 
for afterschool programs offer a number of benefits. 
Measuring Youth Program Quality: A Guide to Assessment 
Tools (Yohalem, Wilson-Ahlstrom, Fischer, & Shinn, 


2009) compares a number of different tools. Its authors 
point out that standardized self-assessment tools can be 
user-friendly and require little training. When training is 
needed, it can often be provided through in-person or 
Internet sessions with the tool’s developers. Using a stan¬ 
dardized tool means that afterschool staff do not need to 
have the expertise necessary to score and interpret the 
assessment. In addition, many standardized tools are free 
or low cost (Yohalem et al, 2009). 

Although they were developed by many different 
researchers and practitioners, the various standardized 
self-assessment tools for afterschool programs share a 
common core of effective practices (Yohalem et al., 
2009). They generally are based in 
youth development principles, 
emphasizing interactions among 
youth and staff. They tend to as¬ 
sess safety, skill-building opportu¬ 
nities, social norms, and program 
routine or structure (Granger et 
al., 2007)—factors that are impor¬ 
tant to students’ overall develop¬ 
ment but may not build academic 
skills. Few self-assessment tools 
include domains representing the 
elements that have been found to improve academic 
achievement, as identified in the IES practice guide 
(Beckett et al., 2009). 

In spring 2010 we conducted a review of nine stan¬ 
dardized self-assessment instruments identified by 
Yohalem and colleagues (2009) to determine which tools 
assess the four recommended practices. The nine tools 
we examined are: 

1. Assessing Afterschool Program Practices Tool by the 
National Institute on Out-of-School Time (2007) 

2. Out-of-School Time Observation Tool by Policy Studies 
Associates (Pechman, 2008) 

3. Program Observation Tool by the National Afterschool 
Association (2010) 

4. Program Quality Observation Scale by Vandell and 
Pierce (2006) 

5. Program Quality Self-Assessment Tool by the New 
York State Afterschool Network (2005) 

6. Promising Practices Rating Scale by the Wisconsin 
Center for Education Research and Policy Studies 
Associates (2005) 

7. Quality Assurance System by Foundations, Inc. (2010) 

8. School-Age Care Environment Rating Scale by the 
Frank Porter Graham Child Development Institute and 
Concordia University, Montreal (2010) 


Few self-assessment tools 
include domains 
representing the elements 
that have been found to 
improve academic 
achievement, as identified 
in the IES practice guide. 
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9.Youth Program Quality Assessment by the David E 
Weikart Center for Youth Program Quality (Smith & 
Hohmann, 2005) 

The review involved analyzing each self-assessment 
tool individually, using a matrix that aligned the domains 
assessed by each instrument tool with the point-of-service 
practices recommended in the IES practice guide (Beckett 
et al., 2009). The review used the following criteria: 

• Alignment with the school day. The tool assesses 
whether afterschool programs used school curricula or 
academic standards and whether the programs involved 
school personnel involved directly or indirectly. 

• Student participation and attendance. The tool in¬ 
cludes items on the number of students participating, 
attendance trends, or whether activities are based on 
student interest. 

• Adapting instruction to individual or small-group 
needs. The tool asks questions on individual or small- 
group tutoring or on adapting instruction to individual 
student needs. 


• Engaging learning experiences. The tool not only 
measures the extent to which youth are engaged but 
also assesses academic content. A number of the tools 
measure “activities” or ’’engagement,” but we coded a 
tool as assessing engaging learning experiences only if 
activities were grounded in academic goals. 

As shown in Table 1, none of the self-assessment 
tools incorporates all four of the point-of-service prac¬ 
tices recommended by IES. Indeed, three of the nine in¬ 
struments we examined do not assess any of the IES 
point-of-service practices. Two tools, SACERS and YPQA, 
address one of the four practices, while two other instru¬ 
ments, OST and PPR, address two of the four. 

Two of the nine self-assessment tools we examined 
address three of the four IES elements. The Afterschool 
Program Practices Tool (APT) from the National Institute 
on Out-of-School Time and the Massachusetts 
Department of Education (2007) incorporates a review 
of the quality of learning activities including homework 
time and targeted academic skill building activities, in 
which youth practice reading, writing, mathematics, sci- 


Table 1. Summary of Self-Assessment Tools and IES Practice Guide Recommendations 



ALIGN THE OST 
PROGRAM 
ACADEMICALLY 
WITH THE 
SCHOOL DAY 

MAXIMIZE 

STUDENT 

PARTICIPATION 

AND 

ATTENDANCE 

ADAPT 

INSTRUCTION TO 
INDIVIDUAL AND 
SMALL-GROUP 
NEEDS 

PROVIDE 

ENGAGING 

LEARNING 

EXPERIENCES 

APT: Assessing Afterschool 
Program Practices Tool 

- 

X 

X 

X 

OST: Out-of-School Time 
Observation Tool 

- 

X 

X 

- 

POT: Program Observation Tool 

- 


- 

- 

PQO: Program Quality 

Observation Scale 

- 

- 

- 

- 

QSA: Program Quality 
Self-Assessment Tool 

X 

X 

. 

X 

PPRS: Promising Practices 

Rating Scale 

- 

X 

- 

X 

QAS: Quality Assurance System 

- 


- 

- 

SACERS: School-Age Care 
Environment Rating Scale 

- 

- 

- 

X 

YPQA: Youth Program 

Quality Assessment 

- 

X 

- 

- 
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ence, and social studies skills, as well as youth participa¬ 
tion and engagement and individualized support. The 
Program Quality Self-Assessment (QSA) Tool by the New 
York State Afterschool Network (2005) focuses on three 
of the recommended elements. It examines the quality of 
learning activities by assessing the 
extent to which activities provide 
academic support including tutor¬ 
ing or homework help, are age- 
appropriate, and are experiential. 

It also assesses youth participation 
and alignment with the school day, 
including links to state and local 
performance benchmarks, connec¬ 
tions with the school curriculum, 
and communication between school 
staff and afterschool staff. 

Making Self-Assessment Work 

Self-assessment, especially assessment of point-of-service 
program practices, can be a powerful tool to provide valu¬ 
able information regarding program quality (Akiva & 
Smith, 2007). Presently, however, none of the available 
self-assessment tools assess the extent to which programs 
implement all of the point-of-service practices geared to¬ 
wards improving academic achievement recommended by 
the 1ES practice guide (Beckett et al., 2009). The next gen¬ 
eration of standardized self-assessment tools for after¬ 
school programs should include evidence-based practices 
found to be effective in improving academic achievement. 

Until enhanced standardized tools are available, af¬ 
terschool programs may want to supplement available 
tools with customized scales that address important prac¬ 
tices shown to increase academic achievement. For guid¬ 
ance in developing customized tools, programs may refer 
to the Program Quality Self-Assessment Tool and the 
Afterschool Program Practices Tool. These tools measure 
a number of the point-of-service program practices rec¬ 
ommended in the IES practice guide (Beckett et al., 2009). 
They may also help program staff understand how to de¬ 
velop relevant items for their customized assessments. 

Homegrown self-assessment tools have their weak¬ 
nesses. They do not have the reliability and validity need¬ 
ed to ensure accurate results, even when based on best- 
practice literature. Although customized self-assessment 
tools may help staff to critically examine their program’s 
strengths and weaknesses, such tools may not accurately 
measure program quality. Caution should be used 
when interpreting results collected with customized 
self-assessment tools, and results should not be used to 


compare one program with another. In addition, users 
and developers of customized instruments should use 
practices that help maximize reliability, such as thoroughly 
training self-assessors and providing scoring rubrics with 
clear standards. 

Self-assessment can be a valu¬ 
able tool to enhance program qual¬ 
ity and facilitate discussions among 
staff about program strengths and 
challenges. Encouraging program 
staff to be observant of and reflec¬ 
tive about key areas of practice at 
the point of service will help en¬ 
sure that programs provide the best 
possible services to participants. 
This process can generate mean¬ 
ingful formative feedback on pro¬ 
gram implementation even if pro¬ 
gram staff customize formalized self-assessment tools. 
The process of self-assessment can promote continual re¬ 
flection and increase the programs ability to help stu¬ 
dents achieve positive academic outcomes. 
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